Evolution of the vacuum Rabi peaks in a detuned atom-cavity system

We measure the transmission spectrum of an optical cavity filled with two-level atoms. For small intensities, the coupling between atoms and a single mode of the cavity splits the spectrum into two normal modes: the vacuum Rabi peaks. We gradually decouple the system in two ways and study the evolution of the vacuum Rabi peaks. First we lower the atom-cavity coupling frequency through its intensity dependence, showing that the peaks shift, deform, exhibit frequency hysteresis, and eventually merge into a single peak that approaches the empty cavity resonance. Second we detune the atoms and the cavity mode and observe an avoided crossing. The normal modes of the coupled system transform into the two resonances of the uncoupled system. We map out the transition region and find good agreement with theoretical predictions. @S1050-2947~97!07910-9#